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The paper "Expert Elicitation for Reliable System 
Design" by Bedford, Quigley and Walls is timely and 
significant for three reasons: 

1. It addresses the importance of expert elicitation 
in systems design and the statistical and practical 
challenges faced when trying to use expert judge- 
ments in a way that is consistent with established 
approaches based on statistical reliability testing. 

2. It rightly focuses our attention on the need for 
a holistic approach to reliability evaluation that 
goes beyond analysis of single projects to also 
include information from "softer" sources such 
as design and operational use. 

3. It recognizes the emerging importance of Bayesian 
methods in providing the "uncertainty calculus" 
to combine evidence from experts with statistical 
reliability data in such a way that system reliabil- 
ity assessments and forecasts can grow and evolve 
as a system changes throughout its life. 

Our own research and experience support many of 
the key thrusts of the authors' ideas. For the last ten 
years we have been applying Bayesian methods — 
more specifically, Bayesian networks (which the au- 
thors refer to in Section 4.2.3) — to a wide variety of 
problem areas (see, e.g., Neil, Malcolm and Shaw, 
2003, and Fenton et al., 2004). This includes system 
dependability evaluation, of which the best known 
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example is the Transport Reliability Assessment Cal- 
culation System (TRACS) (Neil, Fenton, Forey and 
Harris, 2001); this is an early exemplar of the meta 
modeling frameworks cited by the authors in Sec- 
tion 4.1. We have found Bayesian methods to be 
most beneficial to the types of problems mentioned 
by the authors, including the issue of making trade- 
offs between reliability and other system objectives 
like functionality and cost (something we examined 
in detail for software systems in Fenton et al., 2004). 

We have a number of additional observations to 
make about the paper: 

Very often reliability assessments are carried out 
by a client (rather than the design authority) or by 
a procurement agency on behalf of the client. In this 
case, the expert is not the designer but a customer, 
and the impact of this is more general than the au- 
thors appear to suggest in Table 1. Such customers 
may have relevant operational reliability experience 
gained from use of similar products from this or dif- 
ferent suppliers and will, quite correctly, want to 
use this experience to best effect either to reduce 
testing effort or to select suppliers at the procure- 
ment stage. Other situations spring to mind where 
a different perspective would give rise to additional 
problems and challenges, such as COTS (commer- 
cial off the shelf systems). 

There can be a paucity of empirical data for mis- 
sion and safety critical systems simply because the 
systems may be novel or the top events may be 
rare. Probabilistic risk assessment methods aside, 
this problem often forces practitioners to borrow or 
adopt data from different sources, some of uncertain 
provenance, to help make a reliability claim based on 
some structured (or often unstructured) argument. 
Where data do exist, they may only be partially rel- 
evant for a number of reasons. For example, the data 
may be sourced from heterogeneous systems or may 
have been collected under different or uncontrolled 
conditions. Detailed statistical modeling is practi- 
cally and economically infeasible in such "messy" 
situations, but nevertheless judgements have to be 
made. In practice these decisions can be a black art, 
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involving opaque assumptions and unchecked sub- 
jectivity, but in our experience Bayesian methods 
can help bring some rigor and structure. More im- 
portantly, they also encourage transparency and al- 
low uncertainties and assumptions to be modeled 
explicitly. 

In TRACS (Neil, Fenton, Forey and Harris, 2001) 
we built a system that partially or wholly addresses 
some of the authors' aims with some success. Indeed 
the system remains in routine use by QinetiQ to as- 
sess the reliability of military vehicles throughout 
procurement, design, test and operational use. One 
of the original key motivations for TRACS was ex- 
actly the problem identified in Section 4.1 that tra- 
ditional approaches to reliability prediction tend to 
be overly optimistic because they fail to take into 
account design and process factors. The TRACS 
architecture allows estimation of failure rates from 
families of components using a Bayesian hierarchi- 
cal model and then aggregates these into a system 
level reliability distribution, which can then be up- 
dated, using Bayes' rule and likelihood data gath- 
ered at prototype test, system trial and preproduc- 
tion stages. Crucially, at each stage a number of 
expert-based assessments are made to adjust the 
failure rate predictions based on qualitative esti- 
mates of design and manufacturing factors, includ- 
ing subcontractor competence, risk analysis quality, 
design documentation quality, staff reputation and 
skills. A hybrid Bayesian network is then used to 
fuse all of the information to provide a family of 
estimates and predictions throughout system life. 
The state of the art has moved on considerably since 
TRACS, and the Bayesian algorithms used in TRACS 
are now available commercially (AgenaRisk, 2006). 
As a result, model construction is now considerably 
faster and easier than it was when TRACS was first 
implemented in 1999. 

The issue of expert elicitation is becoming increas- 
ingly relevant to extend and supplement six sigma 
approaches. For example, we have recently been work- 
ing with Motorola to help complement their six sigma 
program by using Bayesian methods to represent ex- 
pert judgements about the impact of fundamental 
organizational and process factors on downstream 
product reliability. This is commercially important 
because reliability problems often occur as a result 
of sources of systematic design variability, often it- 
self caused by the ineffective management of out- 
sourced suppliers and problems with communicat- 
ing and implementing system requirements. These 



are issues that are not easily addressed by statistical 
process control techniques, nor are such techniques 
designed to address them, despite their importance. 
Based on this experience, a number of interesting re- 
search issues relevant to the paper spring to mind: 

• Cultural conflict; that is, how do we persuade en- 
gineering experts to express Bayesian priors when 
the dominant culture of statistical process control 
is almost entirely data driven [which can lead to 
what Chapman calls a syndrome of objective ir- 
rationality (Chapman and Ward, 2000)]? 

• What universal organizational and process drivers 
affect what industries and in what way? 

• Can we assess the effects of process factors in 
quantitative terms or encourage the adoption of 
methodical collection and sharing of the necessary 
data? 

The authors implicitly assume that the benefits of 
probability elicitation will only accrue in situations 
where there is already a highly developed reliabil- 
ity methodology to which new techniques can be 
added. In these situations there are already struc- 
ture, methods and data, but what of those who 
need to assess reliability of products sourced from 
less mature organizations or where data collection 
by empirical means is economically infeasible? Here 
elicitation could, perhaps controversially, be used in- 
stead of traditional reliability methods. In this sit- 
uation decisions would turn on "softer" issues, but 
would nevertheless be quantified and the prediction 
ultimately would be verifiable, at least in principle. 

An additional key benefit of probability elicitation 
that was not covered in the paper is that it helps 
codify knowledge, making it available in the future 
for other projects or for other systems. This is im- 
portant because reliability assessment is not just a 
one-off activity undertaken on a single system or 
project or even over the lifetime of such systems; 
it also addresses families of systems that change 
within a changing design organization or usage en- 
vironment. From this perspective, elicitation should 
be seen as a knowledge management opportunity 
rather than as a technical problem to be solved in 
isolation. Such knowledge, if codified and trusted, 
could be reused at reduced cost on future projects 
and used to help communicate engineering judge- 
ment from engineering experts to novices. 

The issue of bias in subjective probability elici- 
tation (which the authors address in Section 3.2) 
has too often been used as an easy excuse not to 
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do Bayesian modeling. We feel strongly that this is- 
sue has been overplayed — a good discussion of this 
can be found in Ayton and Pascoe (1996). More- 
over, in our own work building Bayesian net mod- 
els with domain experts, we have developed a range 
of techniques that minimize the effort required for 
probability elicitation. An example is the use of sim- 
ple predefined distributions that cover most com- 
mon situations that involve ordinal scale variables 
that are conditioned on other ordinal scale variables 
(Fenton and Neil, 2006). 

Finally, we would like to congratulate the authors 
on writing such an interesting, wide ranging and 
thought provoking paper. 
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